{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HTML解析入门及准备URL生成连续技\n",
    "![for humans](https://requests-html.kennethreitz.org/_static/requests-html-logo.png#thumbnail)\n",
    "\n",
    "*  本周主要内容：批量抓取页面基础及技巧\n",
    "*  上周主要内容：HTML解析（parse HTML）及准备URL生成连续技\n",
    "*  20春_Web数据挖掘_week04\n",
    "*  电子讲义设计者：廖汉腾, 许智超\n",
    "<br/>\n",
    "<br/>\n",
    "\n",
    "-----\n",
    "## 复习\n",
    "\n",
    "复习：上周内容，实践\n",
    "\n",
    "* 猎聘PC版 liepin.com 取工作URL参数的牛肉\n",
    "* 如何生成一连串新URL以进一步爬取数据\n",
    "\n",
    "\n",
    "-----\n",
    "## 本周内容及学习目标\n",
    "\n",
    "本周内容聚焦在\n",
    "\n",
    "<mark> 如何有系统的把更多页数据(相同结构)作系统性爬取 </mark>\n",
    "\n",
    "为此，我们需要学习\n",
    "\n",
    "* 翻页：参数字典的拆解\n",
    "  * xpath\n",
    "  * 建构参数模板\n",
    "  * 建构参数字典\n",
    "* 翻页：系统性迭代\n",
    "  * robots.txt\n",
    "  * 频率及时间\n",
    "* 翻页：数据备份与整合\n",
    "  * 储存备份\n",
    "  * 数据整合\n",
    "  \n",
    "### 目标\n",
    "1. 使用 requests-html 爬取并存取网页文字档，查找[requests-html 中文文档](https://cncert.github.io/requests-html-doc-cn/#/)\n",
    "2. 熟悉 [xpath 语法](https://www.w3cschool.cn/xpath/xpath-syntax.html)丶[xpath 节点](https://www.w3cschool.cn/xpath/xpath-nodes.html)\n",
    "3. 使用 [xpath cheatsheet](https://devhints.io/xpath)\n",
    "  * 在 Chrome Inspector 使用\n",
    "  * 在 requests-html (Python) 使用\n",
    "4. 简易使用 [pd.DataFrame](https://www.pypandas.cn/doc/getting_started/dsintro.html#dataframe)\n",
    "5. 参数字典的拆解与迭代\n",
    "6. 翻页数据备份与整合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "/* 本电子讲义使用之CSS */\n",
       "div.code_cell {\n",
       "    background-color: #e5f1fe;\n",
       "}\n",
       "div.cell.selected {\n",
       "    background-color: #effee2;\n",
       "    font-size: 2rem;\n",
       "    line-height: 2.4rem;\n",
       "}\n",
       "div.cell.selected .rendered_html table {\n",
       "    font-size: 2rem !important;\n",
       "    line-height: 2.4rem !important;\n",
       "}\n",
       ".rendered_html pre code {\n",
       "    background-color: #C4E4ff;   \n",
       "    padding: 2px 25px;\n",
       "}\n",
       ".rendered_html pre {\n",
       "    background-color: #99c9ff;\n",
       "}\n",
       "div.code_cell .CodeMirror {\n",
       "    font-size: 2rem !important;\n",
       "    line-height: 2.4rem !important;\n",
       "}\n",
       ".rendered_html img, .rendered_html svg {\n",
       "    max-width: 60%;\n",
       "    height: auto;\n",
       "    float: right;\n",
       "}\n",
       "\n",
       ".rendered_html img[src*=\"#full\"], .rendered_html svg[src*=\"#full\"] {\n",
       "    max-width: 100%;\n",
       "    height: auto;\n",
       "    float: none;\n",
       "}\n",
       "\n",
       ".rendered_html img[src*=\"#thumbnail\"], .rendered_html svg[src*=\"#thumbnail\"] {\n",
       "    max-width: 15%;\n",
       "    height: auto;\n",
       "}\n",
       "\n",
       "/* Gradient transparent - color - transparent */\n",
       "hr {\n",
       "    border: 0;\n",
       "    border-bottom: 1px dashed #ccc;\n",
       "}\n",
       ".emoticon{\n",
       "    font-size: 5rem;\n",
       "    line-height: 4.4rem;\n",
       "    text-align: center;\n",
       "    vertical-align: middle;\n",
       "}\n",
       ".bg-split_apply_comine {\n",
       "    width: 500px;     \n",
       "    height: 300px;\n",
       "    background: url('02_split-apply-comine_500x300.png') -10px -10px;\n",
       "    float: right;\n",
       "}\n",
       ".bg-comine {\n",
       "    width: 175px;\n",
       "    height: 150px;\n",
       "    background: url('02_split-apply-comine_500x300.png') -280px -80px;\n",
       "    float: right;\n",
       "}\n",
       ".bg-apply {\n",
       "    width: 155px;\n",
       "    height: 225px;\n",
       "    background: url('02_split-apply-comine_500x300.png') -160px -30px;\n",
       "    float: right;\n",
       "}\n",
       ".bg-split {\n",
       "    width: 205px;\n",
       "    height: 225px;\n",
       "    background: url('02_split-apply-comine_500x300.png') -10px -30px;\n",
       "    float: right;\n",
       "}\n",
       ".break {\n",
       "                   page-break-after: right; \n",
       "                   width:700px;\n",
       "                   clear:both;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<style>\n",
    "/* 本电子讲义使用之CSS */\n",
    "div.code_cell {\n",
    "    background-color: #e5f1fe;\n",
    "}\n",
    "div.cell.selected {\n",
    "    background-color: #effee2;\n",
    "    font-size: 2rem;\n",
    "    line-height: 2.4rem;\n",
    "}\n",
    "div.cell.selected .rendered_html table {\n",
    "    font-size: 2rem !important;\n",
    "    line-height: 2.4rem !important;\n",
    "}\n",
    ".rendered_html pre code {\n",
    "    background-color: #C4E4ff;   \n",
    "    padding: 2px 25px;\n",
    "}\n",
    ".rendered_html pre {\n",
    "    background-color: #99c9ff;\n",
    "}\n",
    "div.code_cell .CodeMirror {\n",
    "    font-size: 2rem !important;\n",
    "    line-height: 2.4rem !important;\n",
    "}\n",
    ".rendered_html img, .rendered_html svg {\n",
    "    max-width: 60%;\n",
    "    height: auto;\n",
    "    float: right;\n",
    "}\n",
    "\n",
    ".rendered_html img[src*=\"#full\"], .rendered_html svg[src*=\"#full\"] {\n",
    "    max-width: 100%;\n",
    "    height: auto;\n",
    "    float: none;\n",
    "}\n",
    "\n",
    ".rendered_html img[src*=\"#thumbnail\"], .rendered_html svg[src*=\"#thumbnail\"] {\n",
    "    max-width: 15%;\n",
    "    height: auto;\n",
    "}\n",
    "\n",
    "/* Gradient transparent - color - transparent */\n",
    "hr {\n",
    "    border: 0;\n",
    "    border-bottom: 1px dashed #ccc;\n",
    "}\n",
    ".emoticon{\n",
    "    font-size: 5rem;\n",
    "    line-height: 4.4rem;\n",
    "    text-align: center;\n",
    "    vertical-align: middle;\n",
    "}\n",
    ".bg-split_apply_comine {\n",
    "    width: 500px;     \n",
    "    height: 300px;\n",
    "    background: url('02_split-apply-comine_500x300.png') -10px -10px;\n",
    "    float: right;\n",
    "}\n",
    ".bg-comine {\n",
    "    width: 175px;\n",
    "    height: 150px;\n",
    "    background: url('02_split-apply-comine_500x300.png') -280px -80px;\n",
    "    float: right;\n",
    "}\n",
    ".bg-apply {\n",
    "    width: 155px;\n",
    "    height: 225px;\n",
    "    background: url('02_split-apply-comine_500x300.png') -160px -30px;\n",
    "    float: right;\n",
    "}\n",
    ".bg-split {\n",
    "    width: 205px;\n",
    "    height: 225px;\n",
    "    background: url('02_split-apply-comine_500x300.png') -10px -30px;\n",
    "    float: right;\n",
    "}\n",
    ".break {\n",
    "                   page-break-after: right; \n",
    "                   width:700px;\n",
    "                   clear:both;\n",
    "}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 基本模块\n",
    "import pandas as pd\n",
    "from requests_html import HTMLSession"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. 上周整合代码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "edu         5\n",
      "经验         10\n",
      "薪水         72\n",
      "时间         25\n",
      "职称        183\n",
      "公司地点       69\n",
      "公司名称       72\n",
      "链结        193\n",
      "公司URL      72\n",
      "热门公司类型      6\n",
      "dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>职称</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>公司名称</th>\n",
       "      <th>edu</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>阿里巴巴</th>\n",
       "      <th>学历不限</th>\n",
       "      <td>42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>小米</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>明略科技集团</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">华为</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>统招本科</th>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>深圳市客路网络科技有限公司</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>深圳越海全球供应链有限公司</th>\n",
       "      <th>学历不限</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>宝德投资深圳</th>\n",
       "      <th>大专及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>爱回收</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>宝德投资深圳</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>95 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                     职称\n",
       "公司名称          edu      \n",
       "阿里巴巴          学历不限   42\n",
       "小米            统招本科   12\n",
       "明略科技集团        统招本科   11\n",
       "华为            本科及以上  10\n",
       "              统招本科    8\n",
       "...                  ..\n",
       "深圳市客路网络科技有限公司 本科及以上   1\n",
       "深圳越海全球供应链有限公司 学历不限    1\n",
       "宝德投资深圳        大专及以上   1\n",
       "爱回收           本科及以上   1\n",
       "宝德投资深圳        统招本科    1\n",
       "\n",
       "[95 rows x 1 columns]"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 上周C-1B-5 建构 参数模板  \n",
    "参数_compTag_用户体验 = {'中国500强': {'init': ['-1'], 'headckid': ['58d828c357a8cb19'], 'flushckid': ['1'], 'fromSearchBtn': ['2'], 'keyword': ['用户体验'], 'compTag': ['155'], 'ckid': ['58d828c357a8cb19'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'], 'd_ckId': ['6aa779111c1b4ca77cff3648d9dee049'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['6aa779111c1b4ca77cff3648d9dee049']}, '2018互联网300强': {'init': ['-1'], 'headckid': ['58d828c357a8cb19'], 'flushckid': ['1'], 'fromSearchBtn': ['2'], 'keyword': ['用户体验'], 'compTag': ['182'], 'ckid': ['58d828c357a8cb19'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'], 'd_ckId': ['6aa779111c1b4ca77cff3648d9dee049'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['6aa779111c1b4ca77cff3648d9dee049']}, '制造业500强': {'init': ['-1'], 'headckid': ['58d828c357a8cb19'], 'flushckid': ['1'], 'fromSearchBtn': ['2'], 'keyword': ['用户体验'], 'compTag': ['186'], 'ckid': ['58d828c357a8cb19'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'], 'd_ckId': ['6aa779111c1b4ca77cff3648d9dee049'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['6aa779111c1b4ca77cff3648d9dee049']}, 'AI创新成长50强 ': {'init': ['-1'], 'headckid': ['58d828c357a8cb19'], 'flushckid': ['1'], 'fromSearchBtn': ['2'], 'keyword': ['用户体验'], 'compTag': ['189'], 'ckid': ['58d828c357a8cb19'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'], 'd_ckId': ['6aa779111c1b4ca77cff3648d9dee049'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['6aa779111c1b4ca77cff3648d9dee049']}, '独角兽': {'init': ['-1'], 'headckid': ['58d828c357a8cb19'], 'flushckid': ['1'], 'fromSearchBtn': ['2'], 'keyword': ['用户体验'], 'compTag': ['130'], 'ckid': ['58d828c357a8cb19'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'], 'd_ckId': ['6aa779111c1b4ca77cff3648d9dee049'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['6aa779111c1b4ca77cff3648d9dee049']}, '上市公司': {'init': ['-1'], 'headckid': ['58d828c357a8cb19'], 'flushckid': ['1'], 'fromSearchBtn': ['2'], 'keyword': ['用户体验'], 'compTag': ['156'], 'ckid': ['58d828c357a8cb19'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'], 'd_ckId': ['6aa779111c1b4ca77cff3648d9dee049'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['6aa779111c1b4ca77cff3648d9dee049']}}\n",
    "\n",
    "# 上周C-1   多个页面准备测试1 2018互联网300强\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "session = HTMLSession()\n",
    "payload = 参数_compTag_用户体验['2018互联网300强']\n",
    "r = session.get( url, params = payload)\n",
    "\n",
    "# r.url\n",
    "\n",
    "# 上周C-2  简化 A-1   单一页面爬+解析\n",
    "session = HTMLSession()\n",
    "\n",
    "def requests_liepin( url, params):\n",
    "    r = session.get( url , params = payload)\n",
    "\n",
    "    # 先取特定元素, 精准打击其子后辈\n",
    "    主要元素 = r.html.xpath( '//ul[@class=\"sojob-list\"]/li')\n",
    "\n",
    "    # 作为xpath字典，键为我要抓的牛肉名称，值为xpath\n",
    "    dict_xpaths={ \n",
    "        'text': {\n",
    "            'edu':      '//div[contains(@class,\"job-info\")]/p/span[@class=\"edu\"]',\n",
    "            '经验':      '//div[contains(@class,\"job-info\")]/p/span[@class=\"edu\"]/following-sibling::span',\n",
    "            '薪水':    '//div[contains(@class,\"job-info\")]/p/span[@class=\"text-warning\"]', \n",
    "            '时间':    '//div[contains(@class,\"job-info\")]/p/time/@title', \n",
    "            '职称':    '//div[contains(@class,\"job-info\")]/h3/a', \n",
    "            '公司地点': '//div[contains(@class,\"job-info\")]/p/a',\n",
    "            '公司名称': '//div[contains(@class,\"sojob-item-main\")]//p[@class=\"company-name\"]/a', \n",
    "        },\n",
    "        'text_content': {\n",
    "        },\n",
    "        'href': {\n",
    "            '链结':    '//div[contains(@class,\"job-info\")]/h3/a', \n",
    "            '公司URL': '//div[contains(@class,\"sojob-item-main\")]//p[@class=\"company-name\"]/a', \n",
    "        }\n",
    "    }\n",
    "\n",
    "    def get_e_text_content(_xpath_):\n",
    "        # 高级列表推导\n",
    "        暂存结果 = [e.xpath(_xpath_)[0].lxml.text_content() for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    def get_e_text(_xpath_):\n",
    "        # 高级列表推导\n",
    "        暂存结果 = [\"\".join([x.strip() if type(x) is str else x.text.strip() for x in e.xpath(_xpath_)]) for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    def get_e_href(_xpath_):\n",
    "        # 高级列表推导\n",
    "        暂存结果 = [list(e.xpath(_xpath_, first=True).absolute_links)[0] \\\n",
    "                   if len(e.xpath(_xpath_, first=True).absolute_links) >= 1  \\\n",
    "                   else \"\" for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    # 只对主要元素下进行.xpath取值\n",
    "    数据字典 = dict()\n",
    "\n",
    "    数据字典 = {k:get_e_text_content(v) for k,v in dict_xpaths['text_content'].items()}\n",
    "    数据字典.update({k:get_e_text(v) for k,v in dict_xpaths['text'].items()})\n",
    "    数据字典.update({k:get_e_href(v) for k,v in dict_xpaths['href'].items()})\n",
    "\n",
    "    数据 = pd.DataFrame(数据字典)\n",
    "    #数据.to_excel(\"20春_Web数据挖掘_week03_liepin.xlsx\", sheet_name=\"搜查结果\")\n",
    "    return (数据)\n",
    "\n",
    "\n",
    "# 上周C-3   多个页面\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "\n",
    "list_df = list()\n",
    "for k,v in 参数_compTag_用户体验.items():\n",
    "    payload = v\n",
    "    df = requests_liepin( url, params = payload)\n",
    "    df = df.assign (热门公司类型 = k)    \n",
    "    list_df.append(df)\n",
    "\n",
    "df_all = pd.concat(list_df)\n",
    "df_all\n",
    "\n",
    "# 上周C-4   输出\n",
    "df_all.to_excel(\"20春_Web数据挖掘_week03_liepin_各热门公司类型.xlsx\", sheet_name=\"搜查结果\")\n",
    "\n",
    "# 上周C-5 Pandas  基本能力\n",
    "\n",
    "print (df_all.nunique())\n",
    "df_all[['edu']].drop_duplicates()\n",
    "\n",
    "df_all.groupby(['公司名称','edu']).agg({\"职称\":\"count\"}).sort_values(by='职称', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "-----\n",
    "\n",
    "## 本周实践目标\n",
    "<mark> 如何有系统的把更多页数据(相同结构)作系统性爬取 </mark>[猎聘PC版](https://www.liepin.com/zhaopin/)\n",
    "* 翻页：参数字典的拆解\n",
    "  * xpath解析翻页a/@href\n",
    "  * 建构参数模板\n",
    "  * 建构参数字典\n",
    "* 翻页：系统性迭代\n",
    "  * robots.txt\n",
    "  * 频率及时间\n",
    "* 翻页：数据备份与整合\n",
    "  * 储存备份\n",
    "  * 数据整合"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 翻页：参数字典的拆解\n",
    "## xpath解析翻页a/@href"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A-0   单一页面\n",
    "url = \"https://www.liepin.com/zhaopin/?keyword=UX\"\n",
    "session = HTMLSession()\n",
    "r = session.get( url )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<Element 'a' href='/zhaopin/?init=-1&headckid=cfb4b44f7882ec41&fromSearchBtn=2&keyword=UX&ckid=cfb4b44f7882ec41°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=60d7e1a5ab6c4aaec919347248dd3d2c&d_curPage=0&d_pageSize=40&d_headId=60d7e1a5ab6c4aaec919347248dd3d2c&curPage=1'>, <Element 'a' href='/zhaopin/?init=-1&headckid=cfb4b44f7882ec41&fromSearchBtn=2&keyword=UX&ckid=cfb4b44f7882ec41°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=60d7e1a5ab6c4aaec919347248dd3d2c&d_curPage=0&d_pageSize=40&d_headId=60d7e1a5ab6c4aaec919347248dd3d2c&curPage=2'>, <Element 'a' href='/zhaopin/?init=-1&headckid=cfb4b44f7882ec41&fromSearchBtn=2&keyword=UX&ckid=cfb4b44f7882ec41°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=60d7e1a5ab6c4aaec919347248dd3d2c&d_curPage=0&d_pageSize=40&d_headId=60d7e1a5ab6c4aaec919347248dd3d2c&curPage=3'>, <Element 'a' href='/zhaopin/?init=-1&headckid=cfb4b44f7882ec41&fromSearchBtn=2&keyword=UX&ckid=cfb4b44f7882ec41°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=60d7e1a5ab6c4aaec919347248dd3d2c&d_curPage=0&d_pageSize=40&d_headId=60d7e1a5ab6c4aaec919347248dd3d2c&curPage=4'>, <Element 'a' href='/zhaopin/?init=-1&headckid=cfb4b44f7882ec41&fromSearchBtn=2&keyword=UX&ckid=cfb4b44f7882ec41°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=60d7e1a5ab6c4aaec919347248dd3d2c&d_curPage=0&d_pageSize=40&d_headId=60d7e1a5ab6c4aaec919347248dd3d2c&curPage=1'>, <Element 'a' class=('last',) href='/zhaopin/?init=-1&headckid=cfb4b44f7882ec41&fromSearchBtn=2&keyword=UX&ckid=cfb4b44f7882ec41°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=60d7e1a5ab6c4aaec919347248dd3d2c&d_curPage=0&d_pageSize=40&d_headId=60d7e1a5ab6c4aaec919347248dd3d2c&curPage=9' title='末页'>]\n",
      "{'2': '/zhaopin/?init=-1&headckid=cfb4b44f7882ec41&fromSearchBtn=2&keyword=UX&ckid=cfb4b44f7882ec41°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=60d7e1a5ab6c4aaec919347248dd3d2c&d_curPage=0&d_pageSize=40&d_headId=60d7e1a5ab6c4aaec919347248dd3d2c&curPage=1', '3': '/zhaopin/?init=-1&headckid=cfb4b44f7882ec41&fromSearchBtn=2&keyword=UX&ckid=cfb4b44f7882ec41°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=60d7e1a5ab6c4aaec919347248dd3d2c&d_curPage=0&d_pageSize=40&d_headId=60d7e1a5ab6c4aaec919347248dd3d2c&curPage=2', '4': '/zhaopin/?init=-1&headckid=cfb4b44f7882ec41&fromSearchBtn=2&keyword=UX&ckid=cfb4b44f7882ec41°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=60d7e1a5ab6c4aaec919347248dd3d2c&d_curPage=0&d_pageSize=40&d_headId=60d7e1a5ab6c4aaec919347248dd3d2c&curPage=3', '5': '/zhaopin/?init=-1&headckid=cfb4b44f7882ec41&fromSearchBtn=2&keyword=UX&ckid=cfb4b44f7882ec41°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=60d7e1a5ab6c4aaec919347248dd3d2c&d_curPage=0&d_pageSize=40&d_headId=60d7e1a5ab6c4aaec919347248dd3d2c&curPage=4', '下一页': '/zhaopin/?init=-1&headckid=cfb4b44f7882ec41&fromSearchBtn=2&keyword=UX&ckid=cfb4b44f7882ec41°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=60d7e1a5ab6c4aaec919347248dd3d2c&d_curPage=0&d_pageSize=40&d_headId=60d7e1a5ab6c4aaec919347248dd3d2c&curPage=1', '': '/zhaopin/?init=-1&headckid=cfb4b44f7882ec41&fromSearchBtn=2&keyword=UX&ckid=cfb4b44f7882ec41°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=60d7e1a5ab6c4aaec919347248dd3d2c&d_curPage=0&d_pageSize=40&d_headId=60d7e1a5ab6c4aaec919347248dd3d2c&curPage=9'}\n"
     ]
    }
   ],
   "source": [
    "# A-1  xpath 解析翻页a/@href\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a' # 有disabled, current等href是javascript\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a[starts-with(@href,\"/zhaopin\")]'\n",
    "print (r.html.xpath(xpath_翻页a)) # 物件\n",
    "\n",
    "href_列表 = [x.xpath('//@href')[0] for x in r.html.xpath(xpath_翻页a)]\n",
    "#print (href_列表) ，xpath('//@href')[0]\n",
    "#print (href_列表)\n",
    "\n",
    "文字_列表 = [x.text for x in r.html.xpath(xpath_翻页a)]\n",
    "#print (文字_列表) ，text\n",
    "#print (文字_列表)\n",
    "\n",
    "href_字典 = {x.text:x.xpath('//@href')[0]  for x in r.html.xpath(xpath_翻页a)}\n",
    "#print (href_字典) ，text:x.xpath('//@href')[0]\n",
    "print (href_字典)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# r.html.xpath(xpath_翻页a))，# 物件\n",
    "```\n",
    "[<Element 'a' href='/zhaopin/?init=-1&headckid=e528696cb7c0a93d&fromSearchBtn=2&keyword=UX&ckid=e528696cb7c0a93d°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=532a3a7302c6118037ddef6584423497&d_curPage=0&d_pageSize=40&d_headId=532a3a7302c6118037ddef6584423497&curPage=1'>, <Element 'a' href='/zhaopin/?init=-1&headckid=e528696cb7c0a93d&fromSearchBtn=2&keyword=UX&ckid=e528696cb7c0a93d°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=532a3a7302c6118037ddef6584423497&d_curPage=0&d_pageSize=40&d_headId=532a3a7302c6118037ddef6584423497&curPage=2'>, <Element 'a' href='/zhaopin/?init=-1&headckid=e528696cb7c0a93d&fromSearchBtn=2&keyword=UX&ckid=e528696cb7c0a93d°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=532a3a7302c6118037ddef6584423497&d_curPage=0&d_pageSize=40&d_headId=532a3a7302c6118037ddef6584423497&curPage=3'>, <Element 'a' href='/zhaopin/?init=-1&headckid=e528696cb7c0a93d&fromSearchBtn=2&keyword=UX&ckid=e528696cb7c0a93d°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=532a3a7302c6118037ddef6584423497&d_curPage=0&d_pageSize=40&d_headId=532a3a7302c6118037ddef6584423497&curPage=4'>, <Element 'a' href='/zhaopin/?init=-1&headckid=e528696cb7c0a93d&fromSearchBtn=2&keyword=UX&ckid=e528696cb7c0a93d°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=532a3a7302c6118037ddef6584423497&d_curPage=0&d_pageSize=40&d_headId=532a3a7302c6118037ddef6584423497&curPage=1'>, <Element 'a' class=('last',) href='/zhaopin/?init=-1&headckid=e528696cb7c0a93d&fromSearchBtn=2&keyword=UX&ckid=e528696cb7c0a93d°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=532a3a7302c6118037ddef6584423497&d_curPage=0&d_pageSize=40&d_headId=532a3a7302c6118037ddef6584423497&curPage=9' title='末页'>]\n",
    "```\n",
    "\n",
    "# href_列表，输出结果\n",
    "```\n",
    "['/zhaopin/?init=-1&headckid=e528696cb7c0a93d&fromSearchBtn=2&keyword=UX&ckid=e528696cb7c0a93d°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=532a3a7302c6118037ddef6584423497&d_curPage=0&d_pageSize=40&d_headId=532a3a7302c6118037ddef6584423497&curPage=1', '/zhaopin/?init=-1&headckid=e528696cb7c0a93d&fromSearchBtn=2&keyword=UX&ckid=e528696cb7c0a93d°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=532a3a7302c6118037ddef6584423497&d_curPage=0&d_pageSize=40&d_headId=532a3a7302c6118037ddef6584423497&curPage=2', '/zhaopin/?init=-1&headckid=e528696cb7c0a93d&fromSearchBtn=2&keyword=UX&ckid=e528696cb7c0a93d°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=532a3a7302c6118037ddef6584423497&d_curPage=0&d_pageSize=40&d_headId=532a3a7302c6118037ddef6584423497&curPage=3', '/zhaopin/?init=-1&headckid=e528696cb7c0a93d&fromSearchBtn=2&keyword=UX&ckid=e528696cb7c0a93d°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=532a3a7302c6118037ddef6584423497&d_curPage=0&d_pageSize=40&d_headId=532a3a7302c6118037ddef6584423497&curPage=4', '/zhaopin/?init=-1&headckid=e528696cb7c0a93d&fromSearchBtn=2&keyword=UX&ckid=e528696cb7c0a93d°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=532a3a7302c6118037ddef6584423497&d_curPage=0&d_pageSize=40&d_headId=532a3a7302c6118037ddef6584423497&curPage=1', '/zhaopin/?init=-1&headckid=e528696cb7c0a93d&fromSearchBtn=2&keyword=UX&ckid=e528696cb7c0a93d°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=532a3a7302c6118037ddef6584423497&d_curPage=0&d_pageSize=40&d_headId=532a3a7302c6118037ddef6584423497&curPage=9']\n",
    "```\n",
    "\n",
    "# 文字_列表，输出结果\n",
    "```\n",
    "['2', '3', '4', '5', '下一页', '']\n",
    "```\n",
    "\n",
    "# href_字典，输出结果\n",
    "```\n",
    "{'2': '/zhaopin/?init=-1&headckid=e528696cb7c0a93d&fromSearchBtn=2&keyword=UX&ckid=e528696cb7c0a93d°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=532a3a7302c6118037ddef6584423497&d_curPage=0&d_pageSize=40&d_headId=532a3a7302c6118037ddef6584423497&curPage=1', '3': '/zhaopin/?init=-1&headckid=e528696cb7c0a93d&fromSearchBtn=2&keyword=UX&ckid=e528696cb7c0a93d°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=532a3a7302c6118037ddef6584423497&d_curPage=0&d_pageSize=40&d_headId=532a3a7302c6118037ddef6584423497&curPage=2', '4': '/zhaopin/?init=-1&headckid=e528696cb7c0a93d&fromSearchBtn=2&keyword=UX&ckid=e528696cb7c0a93d°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=532a3a7302c6118037ddef6584423497&d_curPage=0&d_pageSize=40&d_headId=532a3a7302c6118037ddef6584423497&curPage=3', '5': '/zhaopin/?init=-1&headckid=e528696cb7c0a93d&fromSearchBtn=2&keyword=UX&ckid=e528696cb7c0a93d°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=532a3a7302c6118037ddef6584423497&d_curPage=0&d_pageSize=40&d_headId=532a3a7302c6118037ddef6584423497&curPage=4', '下一页': '/zhaopin/?init=-1&headckid=e528696cb7c0a93d&fromSearchBtn=2&keyword=UX&ckid=e528696cb7c0a93d°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=532a3a7302c6118037ddef6584423497&d_curPage=0&d_pageSize=40&d_headId=532a3a7302c6118037ddef6584423497&curPage=1', '': '/zhaopin/?init=-1&headckid=e528696cb7c0a93d&fromSearchBtn=2&keyword=UX&ckid=e528696cb7c0a93d°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=532a3a7302c6118037ddef6584423497&d_curPage=0&d_pageSize=40&d_headId=532a3a7302c6118037ddef6584423497&curPage=9'}\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 小结1：所输出的结果为，页面底部的所显示的翻页页数，但不包含所在页页数。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 观察：\n",
    "此网页是否给出开始丶步进丶及结束的信息，以方便我们完成迭代设置\n",
    "\n",
    "* 老问题 URL太长，用上周的URL+query参数解析与pandas数据框找到异同之处\n",
    "* 老问题 怎麽系统化出URL？用上周的URL+query参数解析与pandas数据框找到异同之处的时候，顺便构建参数字典，至少让以下参数可调\n",
    "  * 搜索关键词：上周keyword\n",
    "  * 页码在哪？\n",
    "* 实践挑战：如何把上周代码模块化为我们所用？\n",
    "\n",
    "-----\n",
    "\n",
    "## 建构参数模板\n",
    "\n",
    "```python\n",
    "\n",
    "# 上周B-1 使用 urllib.parse 解析\n",
    "from urllib.parse import urlparse, parse_qs\n",
    "\n",
    "\n",
    "# 上周B-2 使用 pd.DataFrame进行 unuinque()相异值计量比对 \n",
    "import pandas as pd\n",
    "df = pd.DataFrame([ urlparse(x) for x in 公司数据选择器链结.values()])\n",
    "print(df.nunique())\n",
    "\n",
    "# 上周B-3 针对query 再解析之 \n",
    "#df_qs = pd.DataFrame([ parse_qs(x) for x in df['query'] ])\n",
    "df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df['query'] ])\n",
    "print(df.nunique())\n",
    "\n",
    "# 上周B-4 建构 参数模板 及 字典_compTag\n",
    "def parse_url_qs_for_compTag (url):\n",
    "    six_parts = urlparse(url) \n",
    "    out = parse_qs(six_parts.query)\n",
    "    return (out)\n",
    "\n",
    "# parse_url_qs_for_compTag(list(公司数据选择器链结.values())[0])['compTag']\n",
    "参数模板 = parse_url_qs_for_compTag(list(公司数据选择器链结.values())[0])\n",
    "print(参数模板)\n",
    "# [ parse_url_qs_for_compTag(x)['compTag'] for x in 公司数据选择器链结.values()]\n",
    "[ parse_url_qs_for_compTag(x)['compTag'][0] for x in 公司数据选择器链结.values()]\n",
    "\n",
    "字典_compTag = { k:parse_url_qs_for_compTag(v)['compTag'][0] for k,v in 公司数据选择器链结.items()}\n",
    "print (字典_compTag)\n",
    "\n",
    "# B-5 建构 参数模板  \n",
    "def 参数模板生成(compTag , keyword ):\n",
    "    参数 = 参数模板.copy()\n",
    "    参数['compTag'] = compTag\n",
    "    参数['keyword'] = keyword\n",
    "    return (参数)\n",
    "\n",
    "参数_compTag_用户体验 = { k:参数模板生成(compTag = [v], keyword = ['用户体验']) for k,v in 字典_compTag.items()}\n",
    "print(参数_compTag_用户体验)\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>scheme</th>\n",
       "      <th>netloc</th>\n",
       "      <th>path</th>\n",
       "      <th>params</th>\n",
       "      <th>query</th>\n",
       "      <th>fragment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/zhaopin/</td>\n",
       "      <td></td>\n",
       "      <td>init=-1&amp;headckid=cfb4b44f7882ec41&amp;fromSearchBt...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/zhaopin/</td>\n",
       "      <td></td>\n",
       "      <td>init=-1&amp;headckid=cfb4b44f7882ec41&amp;fromSearchBt...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/zhaopin/</td>\n",
       "      <td></td>\n",
       "      <td>init=-1&amp;headckid=cfb4b44f7882ec41&amp;fromSearchBt...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/zhaopin/</td>\n",
       "      <td></td>\n",
       "      <td>init=-1&amp;headckid=cfb4b44f7882ec41&amp;fromSearchBt...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/zhaopin/</td>\n",
       "      <td></td>\n",
       "      <td>init=-1&amp;headckid=cfb4b44f7882ec41&amp;fromSearchBt...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/zhaopin/</td>\n",
       "      <td></td>\n",
       "      <td>init=-1&amp;headckid=cfb4b44f7882ec41&amp;fromSearchBt...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  scheme netloc       path params  \\\n",
       "0                /zhaopin/          \n",
       "1                /zhaopin/          \n",
       "2                /zhaopin/          \n",
       "3                /zhaopin/          \n",
       "4                /zhaopin/          \n",
       "5                /zhaopin/          \n",
       "\n",
       "                                               query fragment  \n",
       "0  init=-1&headckid=cfb4b44f7882ec41&fromSearchBt...           \n",
       "1  init=-1&headckid=cfb4b44f7882ec41&fromSearchBt...           \n",
       "2  init=-1&headckid=cfb4b44f7882ec41&fromSearchBt...           \n",
       "3  init=-1&headckid=cfb4b44f7882ec41&fromSearchBt...           \n",
       "4  init=-1&headckid=cfb4b44f7882ec41&fromSearchBt...           \n",
       "5  init=-1&headckid=cfb4b44f7882ec41&fromSearchBt...           "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scheme      1\n",
      "netloc      1\n",
      "path        1\n",
      "params      1\n",
      "query       5\n",
      "fragment    1\n",
      "dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>init</th>\n",
       "      <th>headckid</th>\n",
       "      <th>fromSearchBtn</th>\n",
       "      <th>keyword</th>\n",
       "      <th>ckid</th>\n",
       "      <th>siTag</th>\n",
       "      <th>d_sfrom</th>\n",
       "      <th>d_ckId</th>\n",
       "      <th>d_curPage</th>\n",
       "      <th>d_pageSize</th>\n",
       "      <th>d_headId</th>\n",
       "      <th>curPage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-1</td>\n",
       "      <td>cfb4b44f7882ec41</td>\n",
       "      <td>2</td>\n",
       "      <td>UX</td>\n",
       "      <td>cfb4b44f7882ec41°radeFlag=0</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>60d7e1a5ab6c4aaec919347248dd3d2c</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>60d7e1a5ab6c4aaec919347248dd3d2c</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1</td>\n",
       "      <td>cfb4b44f7882ec41</td>\n",
       "      <td>2</td>\n",
       "      <td>UX</td>\n",
       "      <td>cfb4b44f7882ec41°radeFlag=0</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>60d7e1a5ab6c4aaec919347248dd3d2c</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>60d7e1a5ab6c4aaec919347248dd3d2c</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-1</td>\n",
       "      <td>cfb4b44f7882ec41</td>\n",
       "      <td>2</td>\n",
       "      <td>UX</td>\n",
       "      <td>cfb4b44f7882ec41°radeFlag=0</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>60d7e1a5ab6c4aaec919347248dd3d2c</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>60d7e1a5ab6c4aaec919347248dd3d2c</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-1</td>\n",
       "      <td>cfb4b44f7882ec41</td>\n",
       "      <td>2</td>\n",
       "      <td>UX</td>\n",
       "      <td>cfb4b44f7882ec41°radeFlag=0</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>60d7e1a5ab6c4aaec919347248dd3d2c</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>60d7e1a5ab6c4aaec919347248dd3d2c</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-1</td>\n",
       "      <td>cfb4b44f7882ec41</td>\n",
       "      <td>2</td>\n",
       "      <td>UX</td>\n",
       "      <td>cfb4b44f7882ec41°radeFlag=0</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>60d7e1a5ab6c4aaec919347248dd3d2c</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>60d7e1a5ab6c4aaec919347248dd3d2c</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-1</td>\n",
       "      <td>cfb4b44f7882ec41</td>\n",
       "      <td>2</td>\n",
       "      <td>UX</td>\n",
       "      <td>cfb4b44f7882ec41°radeFlag=0</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>60d7e1a5ab6c4aaec919347248dd3d2c</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>60d7e1a5ab6c4aaec919347248dd3d2c</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  init          headckid fromSearchBtn keyword                         ckid  \\\n",
       "0   -1  cfb4b44f7882ec41             2      UX  cfb4b44f7882ec41°radeFlag=0   \n",
       "1   -1  cfb4b44f7882ec41             2      UX  cfb4b44f7882ec41°radeFlag=0   \n",
       "2   -1  cfb4b44f7882ec41             2      UX  cfb4b44f7882ec41°radeFlag=0   \n",
       "3   -1  cfb4b44f7882ec41             2      UX  cfb4b44f7882ec41°radeFlag=0   \n",
       "4   -1  cfb4b44f7882ec41             2      UX  cfb4b44f7882ec41°radeFlag=0   \n",
       "5   -1  cfb4b44f7882ec41             2      UX  cfb4b44f7882ec41°radeFlag=0   \n",
       "\n",
       "                                           siTag         d_sfrom  \\\n",
       "0  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw  search_unknown   \n",
       "1  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw  search_unknown   \n",
       "2  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw  search_unknown   \n",
       "3  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw  search_unknown   \n",
       "4  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw  search_unknown   \n",
       "5  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw  search_unknown   \n",
       "\n",
       "                             d_ckId d_curPage d_pageSize  \\\n",
       "0  60d7e1a5ab6c4aaec919347248dd3d2c         0         40   \n",
       "1  60d7e1a5ab6c4aaec919347248dd3d2c         0         40   \n",
       "2  60d7e1a5ab6c4aaec919347248dd3d2c         0         40   \n",
       "3  60d7e1a5ab6c4aaec919347248dd3d2c         0         40   \n",
       "4  60d7e1a5ab6c4aaec919347248dd3d2c         0         40   \n",
       "5  60d7e1a5ab6c4aaec919347248dd3d2c         0         40   \n",
       "\n",
       "                           d_headId curPage  \n",
       "0  60d7e1a5ab6c4aaec919347248dd3d2c       1  \n",
       "1  60d7e1a5ab6c4aaec919347248dd3d2c       2  \n",
       "2  60d7e1a5ab6c4aaec919347248dd3d2c       3  \n",
       "3  60d7e1a5ab6c4aaec919347248dd3d2c       4  \n",
       "4  60d7e1a5ab6c4aaec919347248dd3d2c       1  \n",
       "5  60d7e1a5ab6c4aaec919347248dd3d2c       9  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "init             1\n",
      "headckid         1\n",
      "fromSearchBtn    1\n",
      "keyword          1\n",
      "ckid             1\n",
      "siTag            1\n",
      "d_sfrom          1\n",
      "d_ckId           1\n",
      "d_curPage        1\n",
      "d_pageSize       1\n",
      "d_headId         1\n",
      "curPage          5\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# A-2 建构参数模板：找到关键参数及参数结构\n",
    "# URL太长，用上周的URL+query参数解析\n",
    "\n",
    "# 需要模组库\n",
    "from urllib.parse import urlparse, parse_qs  #使用 urllib.parse 解析\n",
    "import pandas as pd\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "# 总体目标：输入 href_列表, 建构出参数字典\n",
    "\n",
    "# urlparse 解析后丢入数据框\n",
    "#使用 pd.DataFrame进行 unuinque()相异值计量比对 \n",
    "df = pd.DataFrame([ urlparse(x) for x in href_列表])\n",
    "#针对query 再解析之 \n",
    "#df_qs = pd.DataFrame([ parse_qs(x) for x in df['query'] ])\n",
    "df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df['query'] ])\n",
    "\n",
    "display(df)\n",
    "print(df.nunique())\n",
    "display(df_qs)\n",
    "print(df_qs.nunique())\n",
    "\n",
    "df_qs.curPage\n",
    "df_qs = df_qs.assign (curPage_int=df_qs.curPage.astype(int)) # 变成整数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 观察：\n",
    "* query\n",
    "* curPage 5次, 最大值9, 本页不算?\n",
    "\n",
    "-----\n",
    "\n",
    "## 建构参数模板：curPage\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'init': ['-1'], 'headckid': ['cfb4b44f7882ec41'], 'fromSearchBtn': ['2'], 'keyword': ['UX'], 'ckid': ['cfb4b44f7882ec41°radeFlag=0'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'], 'd_ckId': ['60d7e1a5ab6c4aaec919347248dd3d2c'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['60d7e1a5ab6c4aaec919347248dd3d2c'], 'curPage': ['1']}\n",
      "{'2': '/zhaopin/?init=-1&headckid=cfb4b44f7882ec41&fromSearchBtn=2&keyword=UX&ckid=cfb4b44f7882ec41°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=60d7e1a5ab6c4aaec919347248dd3d2c&d_curPage=0&d_pageSize=40&d_headId=60d7e1a5ab6c4aaec919347248dd3d2c&curPage=1', '3': '/zhaopin/?init=-1&headckid=cfb4b44f7882ec41&fromSearchBtn=2&keyword=UX&ckid=cfb4b44f7882ec41°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=60d7e1a5ab6c4aaec919347248dd3d2c&d_curPage=0&d_pageSize=40&d_headId=60d7e1a5ab6c4aaec919347248dd3d2c&curPage=2', '4': '/zhaopin/?init=-1&headckid=cfb4b44f7882ec41&fromSearchBtn=2&keyword=UX&ckid=cfb4b44f7882ec41°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=60d7e1a5ab6c4aaec919347248dd3d2c&d_curPage=0&d_pageSize=40&d_headId=60d7e1a5ab6c4aaec919347248dd3d2c&curPage=3', '5': '/zhaopin/?init=-1&headckid=cfb4b44f7882ec41&fromSearchBtn=2&keyword=UX&ckid=cfb4b44f7882ec41°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=60d7e1a5ab6c4aaec919347248dd3d2c&d_curPage=0&d_pageSize=40&d_headId=60d7e1a5ab6c4aaec919347248dd3d2c&curPage=4', '下一页': '/zhaopin/?init=-1&headckid=cfb4b44f7882ec41&fromSearchBtn=2&keyword=UX&ckid=cfb4b44f7882ec41°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=60d7e1a5ab6c4aaec919347248dd3d2c&d_curPage=0&d_pageSize=40&d_headId=60d7e1a5ab6c4aaec919347248dd3d2c&curPage=1', '': '/zhaopin/?init=-1&headckid=cfb4b44f7882ec41&fromSearchBtn=2&keyword=UX&ckid=cfb4b44f7882ec41°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=60d7e1a5ab6c4aaec919347248dd3d2c&d_curPage=0&d_pageSize=40&d_headId=60d7e1a5ab6c4aaec919347248dd3d2c&curPage=9'}\n"
     ]
    }
   ],
   "source": [
    "# A-2 建构参数模板：找到关键参数及参数结构\n",
    "#建构 参数模板 及 字典？\n",
    "def parse_url_qs_for_curPage (url):\n",
    "    six_parts = urlparse(url) \n",
    "    out = parse_qs(six_parts.query)\n",
    "    return (out)\n",
    "\n",
    "# 取一例做模板\n",
    "# parse_url_qs_for_curPage(href_列表[0])？['curPage']？\n",
    "参数模板 = parse_url_qs_for_curPage(href_列表[0])\n",
    "print (参数模板)\n",
    "\n",
    "print (href_字典)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "9\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{0: {'init': ['-1'],\n",
       "  'headckid': ['cfb4b44f7882ec41'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['UX'],\n",
       "  'ckid': ['cfb4b44f7882ec41°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'curPage': [0]},\n",
       " 1: {'init': ['-1'],\n",
       "  'headckid': ['cfb4b44f7882ec41'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['UX'],\n",
       "  'ckid': ['cfb4b44f7882ec41°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'curPage': [1]},\n",
       " 2: {'init': ['-1'],\n",
       "  'headckid': ['cfb4b44f7882ec41'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['UX'],\n",
       "  'ckid': ['cfb4b44f7882ec41°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'curPage': [2]},\n",
       " 3: {'init': ['-1'],\n",
       "  'headckid': ['cfb4b44f7882ec41'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['UX'],\n",
       "  'ckid': ['cfb4b44f7882ec41°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'curPage': [3]},\n",
       " 4: {'init': ['-1'],\n",
       "  'headckid': ['cfb4b44f7882ec41'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['UX'],\n",
       "  'ckid': ['cfb4b44f7882ec41°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'curPage': [4]},\n",
       " 5: {'init': ['-1'],\n",
       "  'headckid': ['cfb4b44f7882ec41'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['UX'],\n",
       "  'ckid': ['cfb4b44f7882ec41°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'curPage': [5]},\n",
       " 6: {'init': ['-1'],\n",
       "  'headckid': ['cfb4b44f7882ec41'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['UX'],\n",
       "  'ckid': ['cfb4b44f7882ec41°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'curPage': [6]},\n",
       " 7: {'init': ['-1'],\n",
       "  'headckid': ['cfb4b44f7882ec41'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['UX'],\n",
       "  'ckid': ['cfb4b44f7882ec41°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'curPage': [7]},\n",
       " 8: {'init': ['-1'],\n",
       "  'headckid': ['cfb4b44f7882ec41'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['UX'],\n",
       "  'ckid': ['cfb4b44f7882ec41°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'curPage': [8]},\n",
       " 9: {'init': ['-1'],\n",
       "  'headckid': ['cfb4b44f7882ec41'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['UX'],\n",
       "  'ckid': ['cfb4b44f7882ec41°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['60d7e1a5ab6c4aaec919347248dd3d2c'],\n",
       "  'curPage': [9]}}"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A-3 建构参数模板生成器：keyword curPage\n",
    "\n",
    "def 参数模板生成(keyword, curPage):\n",
    "    参数 = 参数模板.copy()\n",
    "    参数['curPage'] = curPage\n",
    "    参数['keyword'] = keyword\n",
    "    return (参数)\n",
    "\n",
    "参数_keyword_UX_curPage = { \n",
    "    i:参数模板生成(curPage = [i], \\\n",
    "                  keyword = ['UX']) \\\n",
    "    for i,v in href_字典.items()\\\n",
    "    }\n",
    "\n",
    "# print(参数_keyword_UX_curPage) \n",
    "# 只生成本页有的额外翻页URL, 并没有推估到&curPage=9,也没有这页\n",
    "\n",
    "print (df_qs.curPage_int.min()) # 最小值只有1\n",
    "print (df_qs.curPage_int.max()) # 最大值只有9\n",
    "\n",
    "# 应该是 0 (本页)....9(最大值)\n",
    "\n",
    "参数_keyword_UX_curPage = { \n",
    "    i:参数模板生成(curPage = [i], \\\n",
    "                  keyword = ['UX']) \\\n",
    "    for i in range(0,df_qs.curPage_int.max()+1)\\\n",
    "    }\n",
    "参数_keyword_UX_curPage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 小结2：\n",
    "1. curPage，最大值为9，输出结果：本页（0）到最大值（9）\n",
    "2. href_字典，则只生成本页（0）的额外翻页URL, 并没有推估到&curPage=9,也没有这页（0）；即['2', '3', '4', '5', '下一页', '']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 翻页：系统性迭代\n",
    "\n",
    "## 爬亦有道\n",
    "* robots.txt 站长/网站拥有者给搜索引擎的\"道\"\n",
    "* 频率及时间\n",
    "  * 不要爬太快\n",
    "  * 尽量像\"人\"一样礼貌\n",
    "  * time.sleep\n",
    "  \n",
    "```python\n",
    "\n",
    "# 上周C-3   多个页面\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "\n",
    "list_df = list()\n",
    "for k,v in 参数_compTag_用户体验.items():\n",
    "    payload = v\n",
    "    df = requests_liepin( url, params = payload)\n",
    "    df = df.assign (热门公司类型 = k)    \n",
    "    list_df.append(df)\n",
    "\n",
    "df_all = pd.concat(list_df)\n",
    "df_all\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "# B-1 上周C-2  简化 上上周A-1   单一页面爬+解析\n",
    "session = HTMLSession()\n",
    "\n",
    "def requests_liepin( url, params):\n",
    "    r = session.get( url , params = payload)\n",
    "\n",
    "    # 先取特定元素, 精准打击其子后辈\n",
    "    主要元素 = r.html.xpath( '//ul[@class=\"sojob-list\"]/li')\n",
    "\n",
    "    # 作为xpath字典，键为我要抓的牛肉名称，值为xpath\n",
    "    dict_xpaths={ \n",
    "        'text': {\n",
    "            'edu':      '//div[contains(@class,\"job-info\")]/p/span[@class=\"edu\"]',\n",
    "            '经验':      '//div[contains(@class,\"job-info\")]/p/span[@class=\"edu\"]/following-sibling::span',\n",
    "            '薪水':    '//div[contains(@class,\"job-info\")]/p/span[@class=\"text-warning\"]', \n",
    "            '时间':    '//div[contains(@class,\"job-info\")]/p/time/@title', \n",
    "            '职称':    '//div[contains(@class,\"job-info\")]/h3/a', \n",
    "            '公司地点': '//div[contains(@class,\"job-info\")]/p/a',\n",
    "            '公司名称': '//div[contains(@class,\"sojob-item-main\")]//p[@class=\"company-name\"]/a', \n",
    "        },\n",
    "        'text_content': {\n",
    "        },\n",
    "        'href': {\n",
    "            '链结':    '//div[contains(@class,\"job-info\")]/h3/a', \n",
    "            '公司URL': '//div[contains(@class,\"sojob-item-main\")]//p[@class=\"company-name\"]/a', \n",
    "        }\n",
    "    }\n",
    "\n",
    "    def get_e_text_content(_xpath_):\n",
    "        # 高级列表推导\n",
    "        暂存结果 = [e.xpath(_xpath_)[0].lxml.text_content() for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    def get_e_text(_xpath_):\n",
    "        # 高级列表推导\n",
    "        暂存结果 = [\"\".join([x.strip() if type(x) is str else x.text.strip() for x in e.xpath(_xpath_)]) for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    def get_e_href(_xpath_):\n",
    "        # 高级列表推导\n",
    "        暂存结果 = [list(e.xpath(_xpath_, first=True).absolute_links)[0] \\\n",
    "                   if len(e.xpath(_xpath_, first=True).absolute_links) >= 1  \\\n",
    "                   else \"\" for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    # 只对主要元素下进行.xpath取值\n",
    "    数据字典 = dict()\n",
    "\n",
    "    数据字典 = {k:get_e_text_content(v) for k,v in dict_xpaths['text_content'].items()}\n",
    "    数据字典.update({k:get_e_text(v) for k,v in dict_xpaths['text'].items()})\n",
    "    数据字典.update({k:get_e_href(v) for k,v in dict_xpaths['href'].items()})\n",
    "\n",
    "    数据 = pd.DataFrame(数据字典)\n",
    "    #数据.to_excel(\"20春_Web数据挖掘_week03_liepin.xlsx\", sheet_name=\"搜查结果\")\n",
    "    return (数据)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 爬亦有道- 不要爬太快\n",
    "time.sleep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 4.26 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "time.sleep(3+4*random())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "UsageError: Line magic function `%%time` not found.\n"
     ]
    }
   ],
   "source": [
    "# B-2 多个页面，但放慢脚步 time.sleep\n",
    "%%time\n",
    "\n",
    "import time\n",
    "from random import random\n",
    "\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "\n",
    "list_df = list()\n",
    "for k,v in 参数_keyword_用户体验_curPage.items():\n",
    "    payload = v\n",
    "    df = requests_liepin( url, params = payload)\n",
    "    time.sleep(3+4*random())  #放慢脚步 3-7秒, 平均约5秒\n",
    "    df = df.assign (curPage = k)  # 区分  curPage\n",
    "    list_df.append(df)\n",
    "\n",
    "df_all = pd.concat(list_df).reset_index()\n",
    "df_all.index.name = '序'\n",
    "\n",
    "# 上周C-4   输出\n",
    "df_all.to_excel(\"20春_Web数据挖掘_week04_liepin_翻页.xlsx\",\\\n",
    "                sheet_name=\"用户体验\")\n",
    "\n",
    "# 预估时间: 5秒*10 =50\n",
    "# 预估数量: 40*10 =400"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 多个页面+多个关键词\n",
    "time.sleep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# B-3 多个页面+多个关键词\n",
    "import time\n",
    "from random import random\n",
    "\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a[starts-with(@href,\"/zhaopin\")]'\n",
    "\n",
    "keywords = ['用户体验','UX']\n",
    "list_df = list()\n",
    "\n",
    "## 第一页试探有多长的页面\n",
    "for key in keywords:\n",
    "    payload = 参数模板生成(keyword=[key], curPage=['0'])\n",
    "    df = requests_liepin( url, params = payload)\n",
    "    href_列表 = [x.xpath('//@href')[0] for x in r.html.xpath(xpath_翻页a)]\n",
    "    df = pd.DataFrame([ urlparse(x) for x in href_列表])\n",
    "    df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df['query'] ])\n",
    "    df_qs = df_qs.assign (curPage_int=df_qs.curPage.astype(int)) # 变成整数\n",
    "    长度 = df_qs.curPage_int.max()+1\n",
    "    参数_keyword_X_curPage = { \n",
    "        i:参数模板生成(curPage = [i], \\\n",
    "                      keyword = [key]) \\\n",
    "        for i in range(0,长度)\\\n",
    "        }\n",
    "    #print (参数_keyword_X_curPage)\n",
    "    print (key,长度)\n",
    "    \n",
    "    for k,v in 参数_keyword_X_curPage.items():\n",
    "        payload = v\n",
    "        df = requests_liepin( url, params = payload)\n",
    "        time.sleep(3+4*random())  #放慢脚步 3-7秒, 平均约5秒\n",
    "        df = df.assign (keyword = key)  # 区分  keyword    \n",
    "        df = df.assign (curPage = k)  # 区分  curPage    \n",
    "        list_df.append(df)\n",
    "        \n",
    "df_all = pd.concat(list_df).reset_index()\n",
    "df_all.index.name = '序'\n",
    "\n",
    "df_all.to_excel(\"20春_Web数据挖掘_week04_liepin_翻页.xlsx\",\\\n",
    "                sheet_name=\"_\".join(keywords))\n",
    "# 预估时间: 2*5秒*10 =100\n",
    "# 预估数量: 2*40*10 =800"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 翻页：数据备份与整合\n",
    "多个页面+多个关键词执行时，若怕中断最好把每一页的df内容备份做中继"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# C-1 多个页面+多个关键词\n",
    "import time\n",
    "from random import random\n",
    "\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a[starts-with(@href,\"/zhaopin\")]'\n",
    "\n",
    "keywords = ['用户体验','UX','产品需求','PRD']\n",
    "list_df = list()\n",
    "\n",
    "## 第一页试探有多长的页面\n",
    "for key in keywords:\n",
    "    payload = 参数模板生成(keyword=[key], curPage=['0'])\n",
    "    df = requests_liepin( url, params = payload)\n",
    "    href_列表 = [x.xpath('//@href')[0] for x in r.html.xpath(xpath_翻页a)]\n",
    "    df = pd.DataFrame([ urlparse(x) for x in href_列表])\n",
    "    df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df['query'] ])\n",
    "    df_qs = df_qs.assign (curPage_int=df_qs.curPage.astype(int)) # 变成整数\n",
    "    长度 = df_qs.curPage_int.max()+1\n",
    "    参数_keyword_X_curPage = { \n",
    "        i:参数模板生成(curPage = [i], \\\n",
    "                      keyword = [key]) \\\n",
    "        for i in range(0,长度)\\\n",
    "        }\n",
    "    #print (参数_keyword_X_curPage)\n",
    "    print (key,长度)\n",
    "    \n",
    "    for k,v in 参数_keyword_X_curPage.items():\n",
    "        payload = v\n",
    "        df = requests_liepin( url, params = payload)\n",
    "        time.sleep(3+4*random())  #放慢脚步 3-7秒, 平均约5秒\n",
    "        ## 备份\n",
    "        df.to_csv(\"20春_Web数据挖掘_week04_liepin_{key}_{k}.tsv\"\\\n",
    "                  .format(key=key, k=k), sep=\"\\t\", encoding=\"utf8\")\n",
    "        \n",
    "        df = df.assign (keyword = key)  # 区分  keyword    \n",
    "        df = df.assign (curPage = k)  # 区分  curPage    \n",
    "        list_df.append(df)\n",
    "        \n",
    "df_all = pd.concat(list_df).reset_index()\n",
    "df_all.index.name = '序'\n",
    "\n",
    "df_all.to_excel(\"20春_Web数据挖掘_week04_liepin_翻页_4.xlsx\",\\\n",
    "                sheet_name=\"_\".join(keywords))\n",
    "# 预估时间: 4*5秒*10 =200\n",
    "# 预估数量: 4*40*10 =1600"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 本周练习\n",
    "\n",
    "* 开始试验各类参数的调整\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 翻页：参数字典的拆解\n",
    "* xpath解析翻页a/@href\n",
    "* 建构参数模板\n",
    "* 建构参数字典\n",
    "### 翻页：系统性迭代\n",
    "* robots.txt\n",
    "* 频率及时间\n",
    "### 翻页：数据备份与整合\n",
    "* 储存备份\n",
    "* 数据整合"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 翻页：参数字典的拆解"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "猎聘网，keyword=UX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<Element 'a' class=('first', 'disabled') href='javascript:;' title='首页'>, <Element 'a' class=('disabled',) href='javascript:;'>, <Element 'a' class=('current',) href='javascript:;'>, <Element 'a' href='/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=1'>, <Element 'a' href='/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=2'>, <Element 'a' href='/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=3'>, <Element 'a' href='/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=4'>, <Element 'a' href='/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=1'>, <Element 'a' class=('last',) href='/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=9' title='末页'>]\n"
     ]
    }
   ],
   "source": [
    "# xpath 解析翻页a/@href\n",
    "import requests_html   #引用requests模块\n",
    "from requests_html import HTMLSession  # 引用设立session值\n",
    "\n",
    "url = \"https://www.liepin.com/zhaopin/?keyword=UX\"  # 将页面链接赋值给url，关键词：UX\n",
    "session = HTMLSession()# 设立session\n",
    "r = session.get( url ) # 获取页面信息\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a'\n",
    "print (r.html.xpath(xpath_翻页a))  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<Element 'a' href='/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=1'>, <Element 'a' href='/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=2'>, <Element 'a' href='/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=3'>, <Element 'a' href='/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=4'>, <Element 'a' href='/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=1'>, <Element 'a' class=('last',) href='/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=9' title='末页'>]\n",
      "{'2': '/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=1', '3': '/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=2', '4': '/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=3', '5': '/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=4', '下一页': '/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=1', '': '/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=9'}\n"
     ]
    }
   ],
   "source": [
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a' \n",
    "# 有disabled, current等href是javascript # 找到翻页的xpath\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a[starts-with(@href,\"/zhaopin\")]'\n",
    "# 保留判别式，@href是属性，以/zhaopin结尾的都要\n",
    "print (r.html.xpath(xpath_翻页a)) # 物件  # 将获取的结果打印出来\n",
    "\n",
    "href_列表 = [x.xpath('//@href')[0] for x in r.html.xpath(xpath_翻页a)]\n",
    "# 获取第一个列表元素，设置了for循环\n",
    "#print (href_列表) ，xpath('//@href')[0]\n",
    "#print (href_列表)\n",
    "\n",
    "文字_列表 = [x.text for x in r.html.xpath(xpath_翻页a)]\n",
    "#print (文字_列表) ，text\n",
    "#print (文字_列表)\n",
    "\n",
    "href_字典 = {x.text:x.xpath('//@href')[0]  for x in r.html.xpath(xpath_翻页a)}\n",
    "#print (href_字典) ，text:x.xpath('//@href')[0]\n",
    "print (href_字典)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>scheme</th>\n",
       "      <th>netloc</th>\n",
       "      <th>path</th>\n",
       "      <th>params</th>\n",
       "      <th>query</th>\n",
       "      <th>fragment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/zhaopin/</td>\n",
       "      <td></td>\n",
       "      <td>init=-1&amp;headckid=812cd72aae788222&amp;fromSearchBt...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/zhaopin/</td>\n",
       "      <td></td>\n",
       "      <td>init=-1&amp;headckid=812cd72aae788222&amp;fromSearchBt...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/zhaopin/</td>\n",
       "      <td></td>\n",
       "      <td>init=-1&amp;headckid=812cd72aae788222&amp;fromSearchBt...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/zhaopin/</td>\n",
       "      <td></td>\n",
       "      <td>init=-1&amp;headckid=812cd72aae788222&amp;fromSearchBt...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/zhaopin/</td>\n",
       "      <td></td>\n",
       "      <td>init=-1&amp;headckid=812cd72aae788222&amp;fromSearchBt...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/zhaopin/</td>\n",
       "      <td></td>\n",
       "      <td>init=-1&amp;headckid=812cd72aae788222&amp;fromSearchBt...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  scheme netloc       path params  \\\n",
       "0                /zhaopin/          \n",
       "1                /zhaopin/          \n",
       "2                /zhaopin/          \n",
       "3                /zhaopin/          \n",
       "4                /zhaopin/          \n",
       "5                /zhaopin/          \n",
       "\n",
       "                                               query fragment  \n",
       "0  init=-1&headckid=812cd72aae788222&fromSearchBt...           \n",
       "1  init=-1&headckid=812cd72aae788222&fromSearchBt...           \n",
       "2  init=-1&headckid=812cd72aae788222&fromSearchBt...           \n",
       "3  init=-1&headckid=812cd72aae788222&fromSearchBt...           \n",
       "4  init=-1&headckid=812cd72aae788222&fromSearchBt...           \n",
       "5  init=-1&headckid=812cd72aae788222&fromSearchBt...           "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scheme      1\n",
      "netloc      1\n",
      "path        1\n",
      "params      1\n",
      "query       5\n",
      "fragment    1\n",
      "dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>init</th>\n",
       "      <th>headckid</th>\n",
       "      <th>fromSearchBtn</th>\n",
       "      <th>keyword</th>\n",
       "      <th>ckid</th>\n",
       "      <th>siTag</th>\n",
       "      <th>d_sfrom</th>\n",
       "      <th>d_ckId</th>\n",
       "      <th>d_curPage</th>\n",
       "      <th>d_pageSize</th>\n",
       "      <th>d_headId</th>\n",
       "      <th>curPage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-1</td>\n",
       "      <td>812cd72aae788222</td>\n",
       "      <td>2</td>\n",
       "      <td>UX</td>\n",
       "      <td>812cd72aae788222°radeFlag=0</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>ae6d09cdde92664621eb3986d56bf47d</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>ae6d09cdde92664621eb3986d56bf47d</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1</td>\n",
       "      <td>812cd72aae788222</td>\n",
       "      <td>2</td>\n",
       "      <td>UX</td>\n",
       "      <td>812cd72aae788222°radeFlag=0</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>ae6d09cdde92664621eb3986d56bf47d</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>ae6d09cdde92664621eb3986d56bf47d</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-1</td>\n",
       "      <td>812cd72aae788222</td>\n",
       "      <td>2</td>\n",
       "      <td>UX</td>\n",
       "      <td>812cd72aae788222°radeFlag=0</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>ae6d09cdde92664621eb3986d56bf47d</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>ae6d09cdde92664621eb3986d56bf47d</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-1</td>\n",
       "      <td>812cd72aae788222</td>\n",
       "      <td>2</td>\n",
       "      <td>UX</td>\n",
       "      <td>812cd72aae788222°radeFlag=0</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>ae6d09cdde92664621eb3986d56bf47d</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>ae6d09cdde92664621eb3986d56bf47d</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-1</td>\n",
       "      <td>812cd72aae788222</td>\n",
       "      <td>2</td>\n",
       "      <td>UX</td>\n",
       "      <td>812cd72aae788222°radeFlag=0</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>ae6d09cdde92664621eb3986d56bf47d</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>ae6d09cdde92664621eb3986d56bf47d</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-1</td>\n",
       "      <td>812cd72aae788222</td>\n",
       "      <td>2</td>\n",
       "      <td>UX</td>\n",
       "      <td>812cd72aae788222°radeFlag=0</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>ae6d09cdde92664621eb3986d56bf47d</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>ae6d09cdde92664621eb3986d56bf47d</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  init          headckid fromSearchBtn keyword                         ckid  \\\n",
       "0   -1  812cd72aae788222             2      UX  812cd72aae788222°radeFlag=0   \n",
       "1   -1  812cd72aae788222             2      UX  812cd72aae788222°radeFlag=0   \n",
       "2   -1  812cd72aae788222             2      UX  812cd72aae788222°radeFlag=0   \n",
       "3   -1  812cd72aae788222             2      UX  812cd72aae788222°radeFlag=0   \n",
       "4   -1  812cd72aae788222             2      UX  812cd72aae788222°radeFlag=0   \n",
       "5   -1  812cd72aae788222             2      UX  812cd72aae788222°radeFlag=0   \n",
       "\n",
       "                                           siTag         d_sfrom  \\\n",
       "0  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw  search_unknown   \n",
       "1  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw  search_unknown   \n",
       "2  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw  search_unknown   \n",
       "3  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw  search_unknown   \n",
       "4  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw  search_unknown   \n",
       "5  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw  search_unknown   \n",
       "\n",
       "                             d_ckId d_curPage d_pageSize  \\\n",
       "0  ae6d09cdde92664621eb3986d56bf47d         0         40   \n",
       "1  ae6d09cdde92664621eb3986d56bf47d         0         40   \n",
       "2  ae6d09cdde92664621eb3986d56bf47d         0         40   \n",
       "3  ae6d09cdde92664621eb3986d56bf47d         0         40   \n",
       "4  ae6d09cdde92664621eb3986d56bf47d         0         40   \n",
       "5  ae6d09cdde92664621eb3986d56bf47d         0         40   \n",
       "\n",
       "                           d_headId curPage  \n",
       "0  ae6d09cdde92664621eb3986d56bf47d       1  \n",
       "1  ae6d09cdde92664621eb3986d56bf47d       2  \n",
       "2  ae6d09cdde92664621eb3986d56bf47d       3  \n",
       "3  ae6d09cdde92664621eb3986d56bf47d       4  \n",
       "4  ae6d09cdde92664621eb3986d56bf47d       1  \n",
       "5  ae6d09cdde92664621eb3986d56bf47d       9  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "init             1\n",
      "headckid         1\n",
      "fromSearchBtn    1\n",
      "keyword          1\n",
      "ckid             1\n",
      "siTag            1\n",
      "d_sfrom          1\n",
      "d_ckId           1\n",
      "d_curPage        1\n",
      "d_pageSize       1\n",
      "d_headId         1\n",
      "curPage          5\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# 建构参数模板：找到关键参数及参数结构\n",
    "# URL太长，用上周的URL+query参数解析\n",
    "\n",
    "# 需要模组库\n",
    "from urllib.parse import urlparse, parse_qs  #使用 urllib.parse 解析\n",
    "import pandas as pd\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "# 总体目标：输入 href_列表, 建构出参数字典\n",
    "\n",
    "# urlparse 解析后丢入数据框\n",
    "#使用 pd.DataFrame进行 unuinque()相异值计量比对 \n",
    "df = pd.DataFrame([ urlparse(x) for x in href_列表])\n",
    "#针对query 再解析之 \n",
    "#df_qs = pd.DataFrame([ parse_qs(x) for x in df['query'] ])\n",
    "df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df['query'] ])\n",
    "\n",
    "display(df)\n",
    "print(df.nunique())\n",
    "display(df_qs)\n",
    "print(df_qs.nunique())\n",
    "\n",
    "df_qs.curPage\n",
    "df_qs = df_qs.assign (curPage_int=df_qs.curPage.astype(int)) # 变成整数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'init': ['-1'], 'headckid': ['812cd72aae788222'], 'fromSearchBtn': ['2'], 'keyword': ['UX'], 'ckid': ['812cd72aae788222°radeFlag=0'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'], 'd_ckId': ['ae6d09cdde92664621eb3986d56bf47d'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['ae6d09cdde92664621eb3986d56bf47d'], 'curPage': ['1']}\n",
      "{'2': '/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=1', '3': '/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=2', '4': '/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=3', '5': '/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=4', '下一页': '/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=1', '': '/zhaopin/?init=-1&headckid=812cd72aae788222&fromSearchBtn=2&keyword=UX&ckid=812cd72aae788222°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=ae6d09cdde92664621eb3986d56bf47d&d_curPage=0&d_pageSize=40&d_headId=ae6d09cdde92664621eb3986d56bf47d&curPage=9'}\n"
     ]
    }
   ],
   "source": [
    "# 建构参数模板：找到关键参数及参数结构\n",
    "#建构 参数模板 及 字典？\n",
    "def parse_url_qs_for_curPage (url):\n",
    "    six_parts = urlparse(url) \n",
    "    out = parse_qs(six_parts.query)\n",
    "    return (out)\n",
    "\n",
    "# 取一例做模板\n",
    "# parse_url_qs_for_curPage(href_列表[0])？['curPage']？\n",
    "参数模板 = parse_url_qs_for_curPage(href_列表[0])\n",
    "print (参数模板)\n",
    "\n",
    "print (href_字典)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "58同城招聘，keyword为UX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<Element 'a' class=('current',) herf='javascript:void(0)'>, <Element 'a' href='https://gz.58.com/job/pn2/?keyword=UX'>, <Element 'a' href='https://gz.58.com/job/pn3/?keyword=UX'>, <Element 'a' href='https://gz.58.com/job/pn4/?keyword=UX'>, <Element 'a' href='https://gz.58.com/job/pn5/?keyword=UX'>, <Element 'a' href='https://gz.58.com/job/pn6/?keyword=UX'>, <Element 'a' href='https://gz.58.com/job/pn7/?keyword=UX'>, <Element 'a' href='https://gz.58.com/job/pn8/?keyword=UX'>, <Element 'a' class=('num_sure',) href=''>]\n"
     ]
    }
   ],
   "source": [
    "# 58同城招聘网 # 关键词keyword=UX\n",
    "import requests_html   # 引用requests模块\n",
    "from requests_html import HTMLSession  # 引用设立session值\n",
    "\n",
    "url = \"https://gz.58.com/job/?keyword=UX\"   # 将页面链接赋值给url\n",
    "session = HTMLSession()     # 设立session，解析数据\n",
    "r = session.get( url )  # 获取页面信息\n",
    "xpath_翻页a = '//div[@class=\"pagesout\"]/span/a'# 找到翻页的xpath \n",
    "print (r.html.xpath(xpath_翻页a))     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 翻页：系统性迭代\n",
    "\n",
    "## 爬亦有道\n",
    "* ***robots.txt 站长/网站拥有者给搜索引擎的\"道\"***\n",
    "   + robots协议是互联网爬虫的一项公认的道德规范，它的全称是“网络爬虫排除标准”（robots exclusion protocol），这个协议用来告诉爬虫，哪些页面是可以抓取的，哪些不可以。\n",
    "   + robots协议是“分段”的，一种是User-agent:，另一种是Allow:或Disallow:\n",
    "   + User-agent表示的是爬虫类型\n",
    "   + Allow代表允许被访问，Disallow代表禁止被访问。\n",
    "   + 互联网并非法外之地，和爬虫相关的法律也在建立和完善之中，目前通用的伦理规范就是robots协议，我们在爬取网络中的信息时，应该有意识地去遵守这个协议。\n",
    "* ***频率及时间***\n",
    "  1. ***不要爬太快***\n",
    "    + 服务器会拒绝频率很高的大型爬虫和恶意爬虫，因为这会给服务器带来极大的压力或伤害。\n",
    "  2. ***尽量像\"人\"一样礼貌***\n",
    "    + 就像是两个人在来来往往的相处中，会考虑对方的感受；在互联网的世界中，我们也要考虑一下服务器对爬虫的感受是怎样的。\n",
    "  3. ***time.sleep***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 5.61 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "import time\n",
    "time.sleep(3+4*random())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 0 ns\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "import time\n",
    "from random import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 多个页面，但放慢脚步 time.sleep\n",
    "\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "\n",
    "list_df = list()\n",
    "for k,v in 参数_keyword_UX_curPage.items():\n",
    "    payload = v\n",
    "    df = requests_liepin( url, params = payload)\n",
    "    time.sleep(3+4*random())  #放慢脚步 3-7秒, 平均约5秒\n",
    "    df = df.assign (curPage = k)  # 区分  curPage\n",
    "    list_df.append(df)\n",
    "\n",
    "df_all = pd.concat(list_df).reset_index()\n",
    "df_all.index.name = '序'\n",
    "\n",
    "# 输出\n",
    "df_all.to_excel(\"20春_Web数据挖掘_week04_liepin_翻页.xlsx\",\\\n",
    "                sheet_name=\"用户体验\")\n",
    "\n",
    "# 预估时间: 5秒*10 =50\n",
    "# 预估数量: 40*10 =400"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<function time.sleep>"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## 多个页面+多个关键词\n",
    "time.sleep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# B-3 多个页面+多个关键词\n",
    "import time\n",
    "from random import random\n",
    "\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a[starts-with(@href,\"/zhaopin\")]'\n",
    "\n",
    "keywords = ['用户体验','UX']\n",
    "list_df = list()\n",
    "\n",
    "## 第一页试探有多长的页面\n",
    "for key in keywords:\n",
    "    payload = 参数模板生成(keyword=[key], curPage=['0'])\n",
    "    df = requests_liepin( url, params = payload)\n",
    "    href_列表 = [x.xpath('//@href')[0] for x in r.html.xpath(xpath_翻页a)]\n",
    "    df = pd.DataFrame([ urlparse(x) for x in href_列表])\n",
    "    df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df['query'] ])\n",
    "    df_qs = df_qs.assign (curPage_int=df_qs.curPage.astype(int)) # 变成整数\n",
    "    长度 = df_qs.curPage_int.max()+1\n",
    "    参数_keyword_X_curPage = { \n",
    "        i:参数模板生成(curPage = [i], \\\n",
    "                      keyword = [key]) \\\n",
    "        for i in range(0,长度)\\\n",
    "        }\n",
    "    #print (参数_keyword_X_curPage)\n",
    "    print (key,长度)\n",
    "    \n",
    "    for k,v in 参数_keyword_X_curPage.items():\n",
    "        payload = v\n",
    "        df = requests_liepin( url, params = payload)\n",
    "        time.sleep(3+4*random())  #放慢脚步 3-7秒, 平均约5秒\n",
    "        df = df.assign (keyword = key)  # 区分  keyword    \n",
    "        df = df.assign (curPage = k)  # 区分  curPage    \n",
    "        list_df.append(df)\n",
    "        \n",
    "df_all = pd.concat(list_df).reset_index()\n",
    "df_all.index.name = '序'\n",
    "\n",
    "df_all.to_excel(\"20春_Web数据挖掘_week04_liepin_翻页.xlsx\",\\\n",
    "                sheet_name=\"_\".join(keywords))\n",
    "# 预估时间: 2*5秒*10 =100\n",
    "# 预估数量: 2*40*10 =800"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 翻页：数据备份与整合\n",
    "多个页面+多个关键词执行时，若怕中断最好把每一页的df内容备份做中继"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# C-1 多个页面+多个关键词\n",
    "import time\n",
    "from random import random\n",
    "\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a[starts-with(@href,\"/zhaopin\")]'\n",
    "\n",
    "keywords = ['用户体验','UX','产品需求','PRD']\n",
    "list_df = list()\n",
    "\n",
    "## 第一页试探有多长的页面\n",
    "for key in keywords:\n",
    "    payload = 参数模板生成(keyword=[key], curPage=['0'])\n",
    "    df = requests_liepin( url, params = payload)\n",
    "    href_列表 = [x.xpath('//@href')[0] for x in r.html.xpath(xpath_翻页a)]\n",
    "    df = pd.DataFrame([ urlparse(x) for x in href_列表])\n",
    "    df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df['query'] ])\n",
    "    df_qs = df_qs.assign (curPage_int=df_qs.curPage.astype(int)) # 变成整数\n",
    "    长度 = df_qs.curPage_int.max()+1\n",
    "    参数_keyword_X_curPage = { \n",
    "        i:参数模板生成(curPage = [i], \\\n",
    "                      keyword = [key]) \\\n",
    "        for i in range(0,长度)\\\n",
    "        }\n",
    "    #print (参数_keyword_X_curPage)\n",
    "    print (key,长度)\n",
    "    \n",
    "    for k,v in 参数_keyword_X_curPage.items():\n",
    "        payload = v\n",
    "        df = requests_liepin( url, params = payload)\n",
    "        time.sleep(3+4*random())  #放慢脚步 3-7秒, 平均约5秒\n",
    "        ## 备份\n",
    "        df.to_csv(\"20春_Web数据挖掘_week04_liepin_{key}_{k}.tsv\"\\\n",
    "                  .format(key=key, k=k), sep=\"\\t\", encoding=\"utf8\")\n",
    "        \n",
    "        df = df.assign (keyword = key)  # 区分  keyword    \n",
    "        df = df.assign (curPage = k)  # 区分  curPage    \n",
    "        list_df.append(df)\n",
    "        \n",
    "df_all = pd.concat(list_df).reset_index()\n",
    "df_all.index.name = '序'\n",
    "\n",
    "df_all.to_excel(\"20春_Web数据挖掘_week04_liepin_翻页_4.xlsx\",\\\n",
    "                sheet_name=\"_\".join(keywords))\n",
    "# 预估时间: 4*5秒*10 =200\n",
    "# 预估数量: 4*40*10 =1600"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 小结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 关于（翻页：系统性迭代）和（翻页：数据备份与整合），time.sleep的内容还没有弄懂\n",
    "   + 多个页面，但放慢脚步 time.sleep\n",
    "   + 多个页面+多个关键词\n",
    "   + 多个页面+多个关键词执行时，若怕中断最好把每一页的df内容备份做中继\n",
    "2. 翻页：参数字典的拆解\n",
    "   + xpath解析翻页a/@href\n",
    "   + 建构参数模板：使用 urllib.parse 解析，针对query 再解析之\n",
    "   + 建构参数字典：找到关键参数及参数结构\n",
    "3. 爬亦有道\n",
    "   + 遵守robots协议，“网络爬虫排除标准”（robots exclusion protocol）\n",
    "   + 互联网并非法外之地，和爬虫相关的法律也在建立和完善之中\n",
    "   + 我们在爬取网络中的信息时，应该有意识地去遵守这个协议。\n",
    "* ***频率及时间***\n",
    "  1. ***不要爬太快***，因为这会给服务器带来极大的压力或伤害。\n",
    "  2. ***尽量像\"人\"一样礼貌***\n",
    "  3. ***time.sleep***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 巩固练习\n",
    "+ 这几周学习到了最常见的是对于网站内容的爬取，包括文本、图片、URL、翻页多页抓取信息等等；"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "文本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "新华社日内瓦4月5日电（记者沈忠浩）世界卫生组织5日晚发布的每日疫情报告指出，目前中国从新冠疫情遏制阶段进入缓解阶段，中国在防控疫情中积累了重要经验。\n",
      "报告援引世卫组织驻华代表高登·加莱亚的话说，中国在防控疫情方面积累了非常重要的经验，特别是因地制宜，采取了适合各地不同情况的公共卫生措施，非常有效地控制了疫情传播。\n",
      "加莱亚指出，通过保持社交距离、隔离、加强个人卫生等举措，中国人民为遏制疫情传播做出巨大努力，产生积极成效。\n",
      "加莱亚表示，各国应共同协作分享抗疫经验和疫情信息，互相借鉴学习，采取及时有效的防控措施。\n",
      "世卫组织的数据显示，截至欧洲中部时间5日18时（北京时间6日零时），全球累计确诊病例达1136851例，累计病亡62955例。\n"
     ]
    }
   ],
   "source": [
    "from requests_html import HTMLSession\n",
    "# 使用session\n",
    "session = HTMLSession()\n",
    "r = session.get(\"http://world.people.com.cn/n1/2020/0406/c1002-31662927.html\") #人民网-国际-滚动新闻《世卫组织：中国在防控疫情中积累重要经验》\n",
    "\n",
    "news = r.html.xpath('//*[@id=\"rwb_zw\"]/p') # 文章\n",
    "\n",
    "for new in news:\n",
    "    print(new.text)  #获得文章内容"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "图片（HTML,Markdown）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://www.linefriends.cn/Storage/master/gallery/201912/201912311645014921990.jpg\" alt=\"\">"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "display(HTML('<img src=\"https://www.linefriends.cn/Storage/master/gallery/201912/201912311645014921990.jpg\" alt=\"\">'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "![](https://www.linefriends.cn/Storage/master/gallery/201912/201912311645014921990.jpg)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, Markdown\n",
    "display(Markdown('![](https://www.linefriends.cn/Storage/master/gallery/201912/201912311645014921990.jpg)'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "749px",
    "left": "1125.609375px",
    "top": "110px",
    "width": "281.390625px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
